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SUMMARY  PAGE 


PROBLEM 


Sonarmen  are  constantly  confronted  with  information  presented 
simultaneously  through  different  sensory  modalities.  Verbal  information  is 
one  type  of  input  to  which  they  are  exposed  ,  Little  is  known  regarding  the 
effect  of  undirected  attention  when  verbal  information  is  presented 
siinul taneously  to  the  auditory  and  visual  modality.  The  question  of 
interest  is  how  do  unimodal  and  multimodal  presentation  conditions 
differentially  affect  performance? 

findings 


The  superiority  of  any  particular  modality  depends  upon  the  task 
requirements,  response  demands  and  the  complexity  of  the  stimuli. 

Hul timodal  stimulation  facilitates  the  RT  of  the  slower  unimodal 
'presentation  while  providing  little  if  any  facilitation  to  the  faster 
un  im  o  dt- 1  approach. 

APPLICATION 

Sonarmen  are  exposed  to  various  types  of  information.  Reports  of 
this  study  and  previous  studies  have  shown  that  the  results  are  task 
dependent.  Therefore,  it  is  important  to  determine  how  different  types  of 
stimuli  interact  with  modes  of  presentation  and  affect  sonar  performance. 
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ABSTRACT 


The  effects  of  presenting  words  either  visually,  aurally,  or  in  both 
modalities  at  once  on  simple  and  choice  reaction  time  were  examined  when  the 
words  were  either  related  or  unreleated  to  a  given  category*  Simple  RT  was 
faster  than  choice  RT,  and  RT  to  related  words  was  faster  than  to  unrelated 
words.  Presenting  words  visually  and  aurally  at  the  same  time  did  not 
improve  simple  RT,  but  it  did  improve  choice  RT  when  the  same  word  appeared 
in  both  modalities.  Words  that  were  not  identical  and  not  related  to  a 
given  category  in  the  bimodal  condition  produced  the  longest  reaction  times. 
It  appeared  that  the  simultaneous  presentation  of  meaningful  information  in 
t wo  modalities  is  beneficial  only  when  the  information  is  rendundant. 


iii 


A  great  deal  of  research  has  been  devoted  to  the  study  of  the 
processing  of  stimuli  presented  to  one  sensory  modality* single  modality. 

Less  attention  has  been  given  to  ways  in  which  different  senses  interact, 
although  we  are  constantly  confronted  with  information  presented 
simultaneously  through  several  sensory  modalities*  Changes  in  sensitivity 
or  temporal  response  to  a  given  stimulus  when  it  is  accompanied  by  a 
stimulus  in  another  modality  have  been  often  reported  (1-5).  In  these 
studies  one  stimulus  acts  as  an  accessory  or  priming  stimulus  with  its  onset 
just  prior  to,  or  simultaneous  with,  a  target  stimulus.  For  example, 
reaction  time  improves  when  an  auditory  stimulus  is  presented  simultaneously 
with,  or  in  close  temporal  proximity  to,  a  visual  stimulus  (6-8).  This  has 
been  referred  to  as  bisensory  facilitation. 

Two  models  have  been  proposed  to  explain  the  facilitation  effect  of 
simultaneous  stimulus  presentation  (9).  The  ”energy  summation”  model 
proposes  that  stimulus  energies  of  the  primary  and  accessory  stimuli  are 
combined  in  such  a  way  that  the  total  energy  of  the  stimuli  is  equivalent  to 
that  of  increasing  the  intensity  of  the  primary  stimulus  alone.  The 
"preparation  enhancement”  model  suggests  that  the  accessory  stimulus 
functions  as  an  alerting  cue  (9).  Both  of  these  models  are  based  largely  on 
studies  which  have  employed  simple  stimuli  such  as  lights  and  tones,  but 
have  failed  to  address  the  issue  of  multimodal  processing  of  meaningful 
information. 

Loveless,  Brebner  and  Hamilton  (10)  reviewed  a  number  of  studies  in 
which  two  signals  were  presented  both  of  which  conveyed  relevant  information 
concerning  a  task.  They  suggested  that  bimodal  presentation  of  information 
facilitates  performance  when  the  stimuli  are  functionally  related.  Another 
important  finding  was  that  subjects  can  divide  their  attention  between  two 
modalities  without  a  decrement  in  response  to  either  modality  alone  (10-11). 


There  seems  to  be  growing  evidence  that  individuals  can  process  two 
or  more  meaningful  messages  presented  simultaneously  in  the  visual  and 
auditory  modalities.  For  example,  Lewis  (12)  as  well  as  Sen  and  Posner  (13) 
found  improved  reaction  time  for  attended  words  in  either  modality  when  the 
same  word  was  presented  simultaneously  to  the  unattended  modality. 
Investigators  have  been  concerneZd  with  determining  conditions  under  which 
facilitation  or  inhibition  occur.  Hanson  (14)  found  facilitation  when  words 
in  the  visual  and  auditory  modalities  were  semantically  related  and/or 
redundant;  however,  in  her  study  attention  was  directed  to  one  modality  at  a 
time. 


Submarine  sonarmen  receive  a  great  diversity  of  meaningful 
information  both  visually  and  aurally.  Yet,  little  research  has 
invests  gated  how  multimodal  processing  affects  detection  and  classification 
of  meaningful  signals  in  one  or  both  modalities.  One  such  study,  which  used 
actual  auditory  signals  but  a  simulated  visual  display  (15)  reported  that  a 
combined  presentation  was  generally  better  for  sonar  performance*  However, 
in  a  more  recent  sonar  detection  and  classification  study,  in  which  actual 
sonar  displays  were  used  for  both  modalities,  bimodal  performance  was  not 
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different  from  unimodal  performance  (16). 

Sonarmen  are  exposed  to  various  types  of  meaningful  information  such 
as  acoustic  displays,  alarms,  and  words.  The  operators  receive  large 
amounts  of  visual  information  such  as  text  on  graphic  screens  which  provide 
input  about  a  contact  of  interest.  Additionally,  conversations  are  almost 
constantly  carried  out  between  operators,  the  sonar  supervisor,  and  the 
officer  of  the  deck  in  the  control  room.  Sonarmen  attend  to  multiple  types 
and  modes  of  information  at  the  same  time  and  make  decisions  and  responses 
based  on  this  information.  Yet,  little  is  known  about  the  effects  of  various 
types  of  multimodal  information  on  sonar  performance. 

The  present  study  tested  the  effects  on  simple  and  choice  RT  of 
visual  and  auditory  words  presented  alone  or  simul taneiously  with  attention 
undirected.  We  were  particularly  interested  in  exploring  the  effects  of 
same  and  different  words  in  each  modality,  and  the  effects  of  related  or 
unrelated  words  to  a  given  category.  The  major  question  was  under  what 
conditions  do  facilitation  or  interference  occur? 

Method 


Subj  ects 


Subjects  were  20  male  university  students  who  volunteered  for  the 
study.  Their  ages  ranged  from  17  to  28  years,  with  a  mean  age  of  21.6 
years.  All  subjects  had  normal  hearing,  normal  or  corrected** to~normal 
vision,  and  no  obvious  motor  deficits. 

Apparatus 

Visual  and  auditory  presentation  of  words  was  controlled  by  an  Apple 
II  Plus  computer  and  a  Votrax  voice  synthesizer.  Synchronization  of  the 
onset  of  words  in  the  simultaneous  visual  and  auditory  conditions  and  data 
collection  were  controlled  by  the  Apple  II  Plus  computer.  Headphones 
(Pioneer  SE-500)  were  used  for  the  auditory  presentation  of  words. 

The  subjects  sat  in  a  chair  approximately  100  cm.  from  the  computer 
screen.  They  listened  to  words  at  a  comfortable  listening  intensity  (60  dB) 
through  a  set  of  headphones.  Subjects  responsed  on  a  Koala  Touch  Pad.  The 
pad  was  designed  with  a  home  button  and  one  or  two  buttons  for  the  simple 
and  choice  reaction  time  tasks  respectively. 

Procedure 


There  were  two  parts  of  this  study.  In  part  one,  simple  reaction 

time  was  measured  when  subjects  removed  their  finger  from  a  home  button  and 

pressed  a  response  button  as  soon  as  they  saw  or  heard  a  word.  There  were 
three  conditions:  a)  visual  presentation  of  words;  b)  auditory  presentation 
of  words;  and  c)  simultaneous  visual  and  auditory  presentation  of  words. 
Reaction  time  was  measured  from  word  onset.  The  entire  experiment  was  under 

computer  control.  The  computer  informed  the  subject  at  the  beginning  of 

each  block  of  trials  whether  they  would  see  or  hear  the  words  or  both.  They 
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were  told  to  place  their  hand  on  the  home  button  of  the  Koala  pad  and  as 
soon  as  they  saw  or  heard  a  word,  they  were  to  press  the  "yes"  button  as 
fast  as  they  could  and  then  return  their  hand  to  the  home  button.  They  were 
reminded  not  to  remove  their  hand  from  the  home  button  until  they  saw  or 
heard  a  word.  If  a  subject* s  hand  moved  before  the  words  were  presented  the 
trial  was  not  run.  This  occurred  on  the  average  of  once  per  subject* 

In  part  two  of  the  study,  subjects  again  were  presented  with  words 
in  the  visual,  auditory,  and  simultaneous  conditions.  However,  choice 
reaction  time  was  assessed  as  they  decided  if  the  word  or  words  presented 
were  semantically  related  to  a  given  category*  The  computer  informed  the 
subjects  of  the  type  of  category  (for  example,  "numbers") .  When  the  word 
was  presented,  they  were  to  decide  if  it  belonged  to  that  category.  They 
were  to  press  "yes1*  if  it  did,  and  the  "no"  button  if  it  did  not.  During 
the  simultaneous  condition  the  two  words  were  either  the  same  (identical)  or 
different.  If  one  or  both  of  the  words  belonged  to  the  category,  they  were 
to  push  the  button  marked  "yes*"  If  neither  belonged  to  the  category,  they 
pressed  the  "no"  button.  For  example,  if  the  category  was  "numbers"  and 
they  heard  "dog"  and  saw  "two",  they  would  press  the  "yes"  button. 

Subjects  were  given  15  practice  trials,  five  trials  of  each  of  the 
three  conditions,  prior  to. each  part  of  the  study.  In  each  condition  of 
parts  one  and  two  of  the  study,  words  were  grouped  into  blocks,  each 
containing  one  additional  practice  trial  and  eight  test  trials.  There  were 
eight  test  words  in  each  block  of  the  visual  and  auditory  conditions  and 
twelve  words  in  each  block  of  the  simultaneous  condition.  Subjects  received 
five  blocks  per  condition  in  each  of  the  simple  and  choice  RT  tasks  (15 
blocks,  140  trials  er  task).  Each  part  of  the  study  was  presented 
separately  with  all  subjects  completing  the  simple  reaction  time  conditions 
(part  one)  before  the  choice  reaction  time  conditions  (part  two).  The  order 
of  presentation  of  visual,  auditory,  and  simultaneous  blocks  of  words  was 
randomized  for  each  subject. 

For  each  block  of  trials  the  procedure  was  as  follows:  a)  subjects 
were  told  which  stimulus  condition  was  to  be  presented  and  the  relevant 
category  for  the  block  (part  two  only) ;  b)  subjects  were  given  a  verbal 
ready  signal;  c)  the  experimenter  pressed  a  key  to  initiate  the  block;  d)  a 
variable  intertrial  interval  (time  between  response  and  onset  of  next 
stimulus)  of  400,  600,  800  or  1000  ms  preceded  the  stimuli;  e)  the  practice 
word  was  presented;  f)  the  subject  responded  by  pressing  the  appropriate 
button;  g)  a  variable  intertrial  interval  of  400,  600,  800,  or  1000  ms 
occurred;  h)  the  first  test  trial  occurred;  i)  the  remaining  trials  were 
presented  at  intertrial  intervals  ranging  from  400  to  1000  ms. 

RESULTS 

The  median  RT  for  each  condition  was  calculated  for  each  subject, 
and  these  were  averaged.  Figure  1  shows  the  averaged  median  RTs  in  the 
various  conditions  of  both  tasks.  Simple  RT  was  much  faster  than  choice  RT 
(_F  (1 , 19 )=668. 20 ,  j>  .0001).  Dunn*  s  tD  test  for  multiple  comparisons  was 
employed  to  determine  whether  there  were  significant  differences  between  all 
of  the  various  means.  All  reported  differences  were  significant  at  the  .01 


A 


SIMPLE  RT  TASK 


AUD.  SAME  DIFF.  VISUAL 

CONDITION 


B 

CHOICE  RT  TASK 


1000 

950 

900 

850 

<n 

£ 

800 

750 

UJ 

£ 

700 

H 

650 

600 

5  50 

500 

AUD-  SAME  DIFF. 

CONDITION 


VISUAL 


Fig.  1.  Mean  median  reaction  tiroes  for  each  condition  of  the  simple  (a)  and 
choice  (e)  reaction  time  tasks 
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level.  Auditory  RT  was  fastest  in  the  simple  task,  whereas  visual  RT  was 
fastest  in  the  choice  task*  In  the  simple  task  RTs  under  the  simultaneous 
conditions  were  similar  to  each  other  and  to  visual  RT,  and  all  three  were 
slower  than  auditory  RT.  In  other  words,  rather  than  bimodal  facilitation, 
it  appeared  that  the  visual  stimulus  interfered  with  simple  reaction  time  to 
the  auditory  stimulus.  In  the  choice  task,  subjects  processed  a  visual  word 
faster  than  an  auditory  word.  When  the  same  word  appeared  in  both 
modalities  performance  was  similar  to  the  visual  RT.  When  different  words 
were  presented  in  the  two  modalities  performance  was  similar  to  the  auditory 
RT. 

Figure  2  shows  the  averaged  median  choice  RTs  for  the  three 
conditions  when  the  subjects  had  to  relate  the  words  to  a  category.  Once 
again,  Dunn1 s  tD  test  was  used  to  analyze  for  differences  between  means,  and 
reported  differences  were  significant  at  the  .01  level.  Visual  RT  was 
always  fastest,  and  RT  was  faster  to  related  words  than  unrelated  words. 

When  the  words  were  related  to  a  category,  auditory  and  simultaneous  RT  did 
not  differ;  when  the  words  were  not  related  to  a  category,  simultaneous  RT 
was  faster  than  auditory  RT.  Within  the  simultaneous  condition  choice  RTs 
were  faster  to  related  than  unrelated  words,  and  faster  when  pairs  of  words 
were  the  same  than  when  they  were  different.  Therefore,  when  redundant 
information  was  presented  in  two  modalities  simultaneously  and  pertained  to 
a  single  positive  response,  performance  was  not  significantly  different  from 
the  best  single  modality.  However,  when  bimodal  information  was 
conflicting,  and  not  related  to  a  specific  category,  performance  was 
hampered. 


CHOICE  RT 


CONDITION 

Fiv,.  2.  Kean  choice  reaction  times  for  words  related  and  unrelated  to  a 
category  when  presented  in  visual,  auditory,  or  simultaneous  conditions 
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DISCUSSION 


As  has  been  reported  before  (13,14,17,18)  performance  improved  when  the 
information  presented  together  in  two  modalities  was  identical  or  resulted 
in  the  same  response*  Performance  was  worse  when  the  stimuli  were  different 
and  indicated  conflicting  responses  (12,14,17,18)*  A  consistent  finding  of 
studies  which  have  examined  bimodal  conditions  has  been  that  RT  to  a  simple 
visual  stimulus  is  facilitated  when  an  accessory  auditory  stimulus  is 
presented  simultaneously  or  within  close  temporal  proximity  with  the  visual 
stimulus  (6-8,19-22)*  However*  the  results  have  been  less  clear  when 
auditory  stimuli  were  the  primary  stimuli  and  visual  stimuli  were 
accessories  (6,8,22)*  The  general  pattern  appears  that  RT  to  the  slower  of 
the  unisensory  modes  of  presentation  has  been  facilitated  by  bimodal 
presentations.  Little  or  no  facilitation  has  been  found  relative  to  the 
faster  mode  of  presentation.  This  finding  is  not  consistent  with 
Nickerson’s  energy  summation  model  (9). 

Which  mode  was  faster  or  slower  depended  upon  whether  a  simple  or 
choice  RT  task  was  employed.  These  results  are  in  agreement  with  the 
findings  of  Kobus  et  al.  (16)  who  found  that  whether  or  not  performance  in  a 
sonar  detection  task  improved  depended  on  the  requirements  of  the  task.  The 
pattern  of  results  concerning  RT  to  the  various  modes  of  presentation  in  the 
choice  RT  task  may  be  accounted  for  by  the  temporal  characteristics 
associated  with  presenting  visual  and  auditory  words,  since  subjects  were 
able  to  read  words  more  quickly  than  they  could  be  spoken. 


The  present  results  of  this  undirected  attention  study  are  fairly 
consistent  with  those  of  previous  directed  attention  studies,  at  stimulus 
properties  in  a  task  can  be  differentially  sampled  in  both  modalities,  that 
more  than  one  set  of  information  can  be  processed  into  memory  at  the  same 
time,  and  that  redundant  stimuli  affect  performance  quite  differently  than 
conflicting  stimuli  (11,23)*  Redundant  meaningful  information  in  multiple 
modalities  can  improve  performance  on  some  taskss.  Therefore,  it  is 
important  to  analyze  each  sonar  task  to  determine  whether  multiple  sources 
of  information  help  or  hinder  performance.  The  superiority  of  any 
particular  modality  depends  on  whether  or  not  it  brings  attention  to  those  i 
properties  of  the  stimuli  demanded  by  the  task.  Future  research  will  need 
to  investigate  the  various  types  of  stimuli  to  which  sonarmen  are  exposed 
and  to  determine  how  these  interacting  stimuli  are  processed.  Whether 
facilitation  or  interference  occurs  seems  to  depend  upon  the  task 
requirements,  response  demands,  complexity  of  the  stimuli,  and  the 
compatibility  between  the  attended  modality  and  the  relevance  of  the 
information  presented  to  it. 
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